Searching the itsi_summary index provides a convenient way to access ITSI KPI values in a normal Splunk events search. Alongside this index however is the ITSI Summary Metrics (itsi_summary_metrics) index which as the name suggests is a metrics index and can therefore provide an even more efficient method of accessing the KPI data. The metrics summary index was introduced with ITSI version 4.6.0.

Preview the metrics

Using the “mpreview” command we can take a look at the fields within the metrics:

|mpreview index=itsi_summary_metrics filter="metric_name=alert_value"

We can see the itsi_kpi_id and itsi_service_id are both available.

Reporting Using the ITSI Summary Metrics Index

Depending on the what the KPI represents we can use “mstats” and the sum or avg aggregators to quickly calculate values over a wide time range, likely a much longer time range than available in your splunk data events index.

The following search calculates the average (mean), max and standard deviation of a response time KPI:

| mstats avg(alert_value) as avg_alert_value max(alert_value) as max_alert_value stdev(alert_value) as stdev_alert_value max(alert_level) as max_alert_level where index=itsi_summary_metrics AND itsi_kpi_id=5a13eeaa1a7a9fff317204be

Stats table using ITSI Summary Metrics data
Note that when the above search was executed over a 30 day time window the search completed in under 1 second.

Timecharts using ITSI Summary Metrics Index

The example below shows how the metrics index can be used to quickly create a timechart using the ITSI Summary Metrics Index. We can quickly split the data into time blocks by simply adding a “span=” clause to the end of the search.

| mstats avg(alert_value) as avg_alert_value max(alert_value) as max_alert_value stdev(alert_value) as stdev_alert_value max(alert_level) as max_alert_level where index=itsi_summary_metrics AND itsi_kpi_id=5a13eeaa1a7a9fff317204be is_service_max_severity_event=0 span=4h

ITSI Summary Metrics data feeding a timechart visualisation
Timechart created using ITSI Summary Metrics

Added Bonus

As you are simply searching a metrics index, that happens to store ITSI KPI data, you don’t need to be on the ITSI search head in order to run these searches. You can use these methods on Splunk Core search heads and build the results into Splunk Core dashboards. See this previous post on how that could be useful.

Further Information

Splunk docs provides a reference for the ITSI Metrics Index.


For 2021 we’ve committed to posting a new Splunk tip every week!

If you want to keep up to date on tips like the one above then sign up below:

Subscribe to our newsletter to receive regular updates from iDelta, including news and updates, information on upcoming events, and Splunk tips and tricks from our team of experts. You can also find us on Twitter and LinkedIn.

Subscribe

* indicates required
Posted by:Stuart Robertson

Stuart Robertson is the Consulting Director at iDelta. He is one of the initial founders of iDelta and has worked there since formation in 2001. Stuart holds various certifications in Core Splunk and ITSI. Stuart also holds a Bsc(Hons) in Computing Science from the University of Glasgow.